Determining driving paths for autonomous driving vehicles based on map data

ABSTRACT

An ADV may determine whether there is preexisting map data for an environment or geographical area/location where the ADV is located/travelling. If there is no preexisting data, the ADV may generate map data based on sensor data obtained from one or more sensors of the ADV. The ADV may determine a path for the ADV based on the generated map data. If there is preexisting map data, the ADV may determine a path for the ADV based on the preexisting map data.

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to operatingautonomous driving vehicles. More particularly, embodiments of thedisclosure relate to determining paths or routes for autonomous drivingvehicles.

BACKGROUND

Vehicles operating in an autonomous mode (e.g., driverless) can relieveoccupants, especially the driver, from some driving-relatedresponsibilities. When operating in an autonomous mode, the vehicle cannavigate to various locations using onboard sensors, allowing thevehicle to travel with minimal human interaction or in some caseswithout any passengers.

Motion planning and control are critical operations in autonomousdriving. Particularly, trajectory planning is a critical component in anautonomous driving system. Conventional trajectory planning techniquesrely heavily on high-quality reference lines, which are guidance paths,e.g., a center line of a road, for autonomous driving vehicles, togenerate stable trajectories.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the disclosure are illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 is a block diagram illustrating a networked system according tosome embodiments.

FIG. 2 is a block diagram illustrating an example of an autonomousvehicle according to some embodiments.

FIGS. 3A-3B are block diagrams illustrating an example of a perceptionand planning system used with an autonomous vehicle according to someembodiments.

FIG. 4A is a block diagram illustrating an example of a perceptionmodule according to some embodiments.

FIG. 4B is a block diagram illustrating an example of a planning moduleaccording to some embodiments.

FIG. 5A is a diagram illustrating an example of an autonomous vehicletraveling down a road according to some embodiments.

FIG. 5B is a diagram illustrating an example of an autonomous vehicletraveling down a road according to some embodiments.

FIG. 6 is a flow diagram illustrating an example of process fordetermining a path for an autonomous vehicle according to someembodiments.

FIG. 7 is a block diagram illustrating a data processing systemaccording to some embodiments.

DETAILED DESCRIPTION

Various embodiments and aspects of the disclosures will be describedwith reference to details discussed below, and the accompanying drawingswill illustrate the various embodiments. The following description anddrawings are illustrative of the disclosure and are not to be construedas limiting the disclosure. Numerous specific details are described toprovide a thorough understanding of various embodiments of the presentdisclosure. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments of the present disclosures.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment may be included in at least oneembodiment of the disclosure. The appearances of the phrase “in oneembodiment” in various places in the specification do not necessarilyall refer to the same embodiment.

According to some embodiments, a new method for determining a path foran autonomous driving vehicle (ADV) is utilized. The ADV may determinewhether there is preexisting map data for an environment or geographicalarea/location where the ADV is located/travelling. If there is nopreexisting data, the ADV may generate map data based on sensor dataobtained from one or more sensors of the ADV. The ADV may determine apath for the ADV based on the generated map data. If there ispreexisting map data, the ADV may determine a path for the ADV based onthe preexisting map data.

The embodiments, implementations, examples, etc., described herein allowthe ADV to generate map data based on sensor data generate by and/orobtained from one or more sensors when there is no preexisting map datafor an environment or geographical area/location. This may allow the ADVto use existing hardware, software, firmware, algorithms, functions,methods, techniques, operations, etc., to determine a path for the ADVwhen there is no preexisting map data. For example, instead ofdeveloping additional hardware, software, firmware, algorithms,functions, methods, techniques, operations, etc., to determine a pathfor the ADV without map data or ceasing automatic operation of the ADV,the existing hardware, software, firmware, algorithms, functions,methods, techniques, operations, may be used because the map data isgenerated on the fly when there is no preexisting map data.

FIG. 1 is a block diagram illustrating an autonomous vehicle networkconfiguration according to some embodiments of the disclosure. Referringto FIG. 1 , network configuration 100 includes autonomous vehicle 101that may be communicatively coupled to one or more servers 103-104 overa network 102. Although there is one autonomous vehicle shown, multipleautonomous vehicles may be coupled to each other and/or coupled toservers 103-104 over network 102. Network 102 may be any type ofnetworks such as a local area network (LAN), a wide area network (WAN)such as the Internet, a cellular network, a satellite network, or acombination thereof, wired or wireless. Server(s) 103-104 may be anykind of servers or a cluster of servers, such as Web or cloud servers,application servers, backend servers, or a combination thereof. Servers103-104 may be data analytics servers, content servers, trafficinformation servers, map and point of interest (MPOI) severs, orlocation servers, etc.

An autonomous vehicle refers to a vehicle that may be configured to inan autonomous mode in which the vehicle navigates through an environmentwith little or no input from a driver. Such an autonomous vehicle caninclude a sensor system having one or more sensors that are configuredto detect information about the environment in which the vehicleoperates. The vehicle and its associated controller(s) use the detectedinformation to navigate through the environment. Autonomous vehicle 101can operate in a manual mode, a full autonomous mode, or a partialautonomous mode.

In one embodiment, autonomous vehicle 101 includes, but is not limitedto, perception and planning system 110, vehicle control system 111,wireless communication system 112, user interface system 113, and sensorsystem 115. Autonomous vehicle 101 may further include certain commoncomponents included in ordinary vehicles, such as, an engine, wheels,steering wheel, transmission, etc., which may be controlled by vehiclecontrol system 111 and/or perception and planning system 110 using avariety of communication signals and/or commands, such as, for example,acceleration signals or commands, deceleration signals or commands,steering signals or commands, braking signals or commands, etc.

Components 110-115 may be communicatively coupled to each other via aninterconnect, a bus, a network, or a combination thereof. For example,components 110-115 may be communicatively coupled to each other via acontroller area network (CAN) bus. A CAN bus is a vehicle bus standarddesigned to allow microcontrollers and devices to communicate with eachother in applications without a host computer. It is a message-basedprotocol, designed originally for multiplex electrical wiring withinautomobiles, but is also used in many other contexts.

Referring now to FIG. 2 , in one embodiment, sensor system 115 includes,but it is not limited to, one or more cameras 211, global positioningsystem (GPS) unit 212, inertial measurement unit (IMU) 213, radar unit214, and a light detection and range (LIDAR) unit 215. GPS system 212may include a transceiver operable to provide information regarding theposition of the autonomous vehicle. IMU unit 213 may sense position andorientation changes of the autonomous vehicle based on inertialacceleration. Radar unit 214 may represent a system that utilizes radiosignals to sense objects within the local environment of the autonomousvehicle. In some embodiments, in addition to sensing objects, radar unit214 may additionally sense the speed and/or heading of the objects.LIDAR unit 215 may sense objects in the environment in which theautonomous vehicle is located using lasers. LIDAR unit 215 could includeone or more laser sources, a laser scanner, and one or more detectors,among other system components. Cameras 211 may include one or moredevices to capture images of the environment surrounding the autonomousvehicle. Cameras 211 may be still cameras and/or video cameras. A cameramay be mechanically movable, for example, by mounting the camera on arotating and/or tilting a platform.

Sensor system 115 may further include other sensors, such as, a sonarsensor, an infrared sensor, a steering sensor, a throttle sensor, abraking sensor, and an audio sensor (e.g., microphone). An audio sensormay be configured to capture sound from the environment surrounding theautonomous vehicle. A steering sensor may be configured to sense thesteering angle of a steering wheel, wheels of the vehicle, or acombination thereof. A throttle sensor and a braking sensor sense thethrottle position and braking position of the vehicle, respectively. Insome situations, a throttle sensor and a braking sensor may beintegrated as an integrated throttle/braking sensor.

In one embodiment, vehicle control system 111 includes, but is notlimited to, steering unit 201, throttle unit 202 (also referred to as anacceleration unit), and braking unit 203. Steering unit 201 is to adjustthe direction or heading of the vehicle. Throttle unit 202 is to controlthe speed of the motor or engine that in turn control the speed andacceleration of the vehicle. Braking unit 203 is to decelerate thevehicle by providing friction to slow the wheels or tires of thevehicle. Note that the components as shown in FIG. 2 may be implementedin hardware, software, or a combination thereof.

Referring back to FIG. 1 , wireless communication system 112 is to allowcommunication between autonomous vehicle 101 and external systems, suchas devices, sensors, other vehicles, etc. For example, wirelesscommunication system 112 can wirelessly communicate with one or moredevices directly or via a communication network, such as servers 103-104over network 102. Wireless communication system 112 can use any cellularcommunication network or a wireless local area network (WLAN), e.g.,using WiFi to communicate with another component or system. Wirelesscommunication system 112 could communicate directly with a device (e.g.,a mobile device of a passenger, a display device, a speaker withinvehicle 101), for example, using an infrared link, Bluetooth, etc. Userinterface system 113 may be part of peripheral devices implementedwithin vehicle 101 including, for example, a keyword, a touch screendisplay device, a microphone, and a speaker, etc.

Some or all of the functions of autonomous vehicle 101 may be controlledor managed by perception and planning system 110, especially whenoperating in an autonomous driving mode. Perception and planning system110 includes the necessary hardware (e.g., processor(s), memory,storage) and software (e.g., operating system, planning and routingprograms) to receive information from sensor system 115, control system111, wireless communication system 112, and/or user interface system113, process the received information, plan a route or path from astarting point to a destination point, and then drive vehicle 101 basedon the planning and control information. Alternatively, perception andplanning system 110 may be integrated with vehicle control system 111.

For example, a user as a passenger may specify a starting location and adestination of a trip, for example, via a user interface. Perception andplanning system 110 obtains the trip related data. For example,perception and planning system 110 may obtain location and routeinformation from an MPOI server, which may be a part of servers 103-104.The location server provides location services and the MPOI serverprovides map services and the POIs of certain locations. Alternatively,such location and MPOI information may be cached locally in a persistentstorage device of perception and planning system 110. In someembodiments, the perception and planning system 110 may not have MPOIinformation (e.g., map data). For example, the perception and planningsystem 110 may not have map data for other environments or geographicalareas/locations, the perception and planning system 110 may not have mapdata for an environment or geographical area/location where theautonomous vehicle 101 is currently travelling or located (e.g., theperception and planning system 110 may have map data for one city butmay not have map data for another city). In another example, theperception and planning system 110 may not have any map data or MPOIinformation (e.g., the perception and planning system 110 may not storeany map data).

While autonomous vehicle 101 is moving along the route, perception andplanning system 110 may also obtain real-time traffic information from atraffic information system or server (TIS). Note that servers 103-104may be operated by a third party entity. Alternatively, thefunctionalities of servers 103-104 may be integrated with perception andplanning system 110. Based on the real-time traffic information, MPOIinformation, and location information, as well as real-time localenvironment data detected or sensed by sensor system 115 (e.g.,obstacles, objects, nearby vehicles), perception and planning system 110can plan an optimal route and drive vehicle 101, for example, viacontrol system 111, according to the planned route to reach thespecified destination safely and efficiently.

Server 103 may be a data analytics system to perform data analyticsservices for a variety of clients. In one embodiment, data analyticssystem 103 includes data collector 121 and machine learning engine 122.Data collector 121 collects driving statistics 123 from a variety ofvehicles, either autonomous vehicles or regular vehicles driven by humandrivers. Driving statistics 123 include information indicating thedriving commands (e.g., throttle, brake, steering commands) issued andresponses of the vehicles (e.g., speeds, accelerations, decelerations,directions) captured by sensors of the vehicles at different points intime. Driving statistics 123 may further include information describingthe driving environments at different points in time, such as, forexample, routes (including starting and destination locations), MPOIs,road conditions, weather conditions, etc.

Based on driving statistics 123, machine learning engine 122 generatesor trains a set of rules, algorithms, and/or predictive models 124 for avariety of purposes. For example, a set of quintic polynomial functionsmay be selected and defined with initial coefficients or parameters.Furthermore, a set of constraints may also be defined based on thehardware characteristics such as sensors specification and specificvehicle designs, which may be obtained from the driving statistics 123.

FIGS. 3A and 3B are block diagrams illustrating an example of aperception and planning system used with an autonomous vehicle accordingto some embodiments. System 300 may be implemented as a part ofautonomous vehicle 101 of FIG. 1 including, but is not limited to,perception and planning system 110, control system 111, and sensorsystem 115. Referring to FIGS. 3A-3B, perception and planning system 110includes, but is not limited to, localization module 301, perceptionmodule 302, prediction module 303, decision module 304, planning module305, control module 306, and routing module 307.

Some or all of modules 301-307 may be implemented in software, hardware,or a combination thereof. For example, these modules may be installed inpersistent storage device 352, loaded into memory 351, and executed byone or more processors (not shown). Note that some or all of thesemodules may be communicatively coupled to or integrated with some or allmodules of vehicle control system 111 of FIG. 2 . Some of modules301-307 may be integrated together as an integrated module.

Localization module 301 determines a current location of autonomousvehicle 300 (e.g., leveraging GPS unit 212) and manages any data relatedto a trip or route of a user. Localization module 301 (also referred toas a map and route module) manages any data related to a trip or routeof a user. A user may log in and specify a starting location and adestination of a trip, for example, via a user interface. Localizationmodule 301 communicates with other components of autonomous vehicle 300,such as map and route information 311, to obtain the trip related data.For example, localization module 301 may obtain location and routeinformation from a location server and a map and POI (MPOI) server. Alocation server provides location services and an MPOI server providesmap services and the POIs of certain locations, which may be cached aspart of map and route information 311. While autonomous vehicle 300 ismoving along the route, localization module 301 may also obtainreal-time traffic information from a traffic information system orserver. In one embodiment, the map and route information 311 may havebeen previously stored in the persistent storage device 352. Forexample, the map and route information 311 may have been previouslydownloaded or copied to the persistent storage device 352. In anotherembodiment, the map and route information 311 may be generated as theautonomous vehicle 300 travels through an environment or geographicalarea. For example, if there is no preexisting map data (e.g., there isno map and route information 311 that was previously stored in thepersistence storage device 352) for the environment or geographicallocation/area where the autonomous vehicle 300 is currentlylocated/travelling, the autonomous vehicle 300 may generate map data forthe environment or geographical location/area based on sensor datareceived or processed by the perception module 302, as discussed in moredetail below. The map data may be generated on the fly or while theautonomous vehicle 300 is travelling through an environment orgeographical area.

Based on the sensor data provided by sensor system 115 and localizationinformation obtained by localization module 301, a perception of thesurrounding environment is determined by perception module 302. Theperception information may represent what an ordinary driver wouldperceive surrounding a vehicle in which the driver is driving. Theperception can include the lane configuration (e.g., straight or curvelanes), traffic light signals, a relative position of another vehicle, apedestrian, a building, crosswalk, or other traffic related signs (e.g.,stop signs, yield signs), etc., for example, in a form of an object.

Perception module 302 may include a computer vision system orfunctionalities of a computer vision system to process and analyzeimages captured by one or more cameras in order to identify objectsand/or features in the environment of the autonomous vehicle. Theobjects can include traffic signals, road way boundaries, othervehicles, pedestrians, and/or obstacles, etc. The computer vision systemmay use an object recognition algorithm, video tracking, and othercomputer vision techniques. In some embodiments, the computer visionsystem can map an environment, track objects, and estimate the speed ofobjects, etc. Perception module 302 can also detect objects based onother sensors data provided by other sensors such as a radar and/orLIDAR. In one embodiment, the perception module 302 may generate mapdata for an environment or geographical area/location. For example, ifthe autonomous vehicle is located or travelling in an environment orgeographical area/location and there is no map data for environment orgeographical area/location, the perception module 302 may generate themap data based on sensor data (e.g., image data, video data, radar data,LIDAR data, etc.) obtained or received from one or more sensors (e.g., acamera, radar, LIDAR, etc.), as discussed in more detail below.

For each of the objects, prediction module 303 predicts what the objectwill behave under the circumstances. The prediction is performed basedon the perception data perceiving the driving environment at the pointin time in view of a set of map/route information 311 and traffic rules312. For example, if the object is a vehicle at an opposing directionand the current driving environment includes an intersection, predictionmodule 303 will predict whether the vehicle will likely move straightforward or make a turn. If the perception data indicates that theintersection has no traffic light, prediction module 303 may predictthat the vehicle may have to fully stop prior to entering theintersection. If the perception data indicates that the vehicle iscurrently at a left-turn only lane or a right-turn only lane, predictionmodule 303 may predict that the vehicle will more likely make a leftturn or right turn respectively. In some embodiments, the map/routeinformation 311 for an environment or geographical area/location may begenerated on the fly (e.g., generated by the perception module 302) asthe autonomous vehicle travels through the environment or geographicalarea/location, as discussed in more detail below.

For each of the objects, decision module 304 makes a decision regardinghow to handle the object. For example, for a particular object (e.g.,another vehicle in a crossing route) as well as its metadata describingthe object (e.g., a speed, direction, turning angle), decision module304 decides how to encounter the object (e.g., overtake, yield, stop,pass). Decision module 304 may make such decisions according to a set ofrules such as traffic rules or driving rules 312, which may be stored inpersistent storage device 352.

Routing module 307 is configured to provide one or more routes or pathsfrom a starting point to a destination point. For a given trip from astart location to a destination location, for example, received from auser, routing module 307 obtains map and route information 311 anddetermines all possible routes or paths from the starting location toreach the destination location. In some embodiments, the map and routeinformation 311 may be generated by the perception module 302, asdiscussed in more detail below. Routing module 307 may generate areference line in a form of a topographic map for each of the routes itdetermines from the starting location to reach the destination location.A reference line refers to an ideal route or path without anyinterference from others such as other vehicles, obstacles, or trafficcondition. That is, if there is no other vehicle, pedestrians, orobstacles on the road, an ADV should exactly or closely follows thereference line. The topographic maps are then provided to decisionmodule 304 and/or planning module 305. Decision module 304 and/orplanning module 305 examine all of the possible routes to select andmodify one of the most optimal routes in view of other data provided byother modules such as traffic conditions from localization module 301,driving environment perceived by perception module 302, and trafficcondition predicted by prediction module 303. The actual path or routefor controlling the ADV may be close to or different from the referenceline provided by routing module 307 dependent upon the specific drivingenvironment at the point in time.

Based on a decision for each of the objects perceived, planning module305 plans a path or route for the autonomous vehicle, as well as drivingparameters (e.g., distance, speed, and/or turning angle), using areference line provided by routing module 307 as a basis. That is, for agiven object, decision module 304 decides what to do with the object,while planning module 305 determines how to do it. For example, for agiven object, decision module 304 may decide to pass the object, whileplanning module 305 may determine whether to pass on the left side orright side of the object. Planning and control data is generated byplanning module 305 including information describing how vehicle 300would move in a next moving cycle (e.g., next route/path segment). Forexample, the planning and control data may instruct vehicle 300 to move10 meters at a speed of 30 mile per hour (mph), then change to a rightlane at the speed of 25 mph.

Based on the planning and control data, control module 306 controls anddrives the autonomous vehicle, by sending proper commands or signals tovehicle control system 111, according to a route or path defined by theplanning and control data. The planning and control data includesufficient information to drive the vehicle from a first point to asecond point of a route or path using appropriate vehicle settings ordriving parameters (e.g., throttle, braking, and turning commands) atdifferent points in time along the path or route.

In one embodiment, the planning phase is performed in a number ofplanning cycles, also referred to as command cycles, such as, forexample, in every time interval of 100 milliseconds (ms). For each ofthe planning cycles or command cycles, one or more control commands willbe issued based on the planning and control data. That is, for every 100ms, planning module 305 plans a next route segment or path segment, forexample, including a target position and the time required for the ADVto reach the target position. Alternatively, planning module 305 mayfurther specify the specific speed, direction, and/or steering angle,etc. In one embodiment, planning module 305 plans a route segment orpath segment for the next predetermined period of time such as 5seconds. For each planning cycle, planning module 305 plans a targetposition for the current cycle (e.g., next 5 seconds) based on a targetposition planned in a previous cycle. Control module 306 then generatesone or more control commands (e.g., throttle, brake, steering controlcommands) based on the planning and control data of the current cycle.

Note that decision module 304 and planning module 305 may be integratedas an integrated module. Decision module 304/planning module 305 mayinclude a navigation system or functionalities of a navigation system todetermine a driving path for the autonomous vehicle. For example, thenavigation system may determine a series of speeds and directionalheadings to effect movement of the autonomous vehicle along a path thatsubstantially avoids perceived obstacles while generally advancing theautonomous vehicle along a roadway-based path leading to an ultimatedestination. The destination may be set according to user inputs viauser interface system 113. The navigation system may update the drivingpath dynamically while the autonomous vehicle is in operation. Thenavigation system can incorporate data from a GPS system and one or moremaps (which may be generated by the perception module 302 or may havebeen previously stored/downloaded) so as to determine the driving pathfor the autonomous vehicle.

Decision module 304/planning module 305 may further include a collisionavoidance system or functionalities of a collision avoidance system toidentify, evaluate, and avoid or otherwise negotiate potential obstaclesin the environment of the autonomous vehicle. For example, the collisionavoidance system may effect changes in the navigation of the autonomousvehicle by operating one or more subsystems in control system 111 toundertake swerving maneuvers, turning maneuvers, braking maneuvers, etc.The collision avoidance system may automatically determine feasibleobstacle avoidance maneuvers on the basis of surrounding trafficpatterns, road conditions, etc. The collision avoidance system may beconfigured such that a swerving maneuver is not undertaken when othersensor systems detect vehicles, construction barriers, etc., in theregion adjacent the autonomous vehicle that would be swerved into. Thecollision avoidance system may automatically select the maneuver that isboth available and maximizes safety of occupants of the autonomousvehicle. The collision avoidance system may select an avoidance maneuverpredicted to cause the least amount of acceleration in a passenger cabinof the autonomous vehicle.

Routing module 307 can generate reference routes, for example, from mapinformation such as information of road segments, vehicular lanes ofroad segments, and distances from lanes to curb. For example, a road maybe divided into sections or segments {A, B, and C} to denote three roadsegments. Three lanes of road segment A may be enumerated {A1, A2, andA3}. A reference route is generated by generating reference points alongthe reference route. For example, for a vehicular lane, routing module307 can connect midpoints of two opposing curbs or extremities of thevehicular lane provided by a map data (which may be generated by theperception module 302 or may have been previously stored/downloaded).Based on the midpoints and machine learning data representing collecteddata points of vehicles previously driven on the vehicular lane atdifferent points in time, routing module 307 can calculate the referencepoints by selecting a subset of the collected data points within apredetermined proximity of the vehicular lane and applying a smoothingfunction to the midpoints in view of the subset of collected datapoints.

Based on reference points or lane reference points, routing module 307may generate a reference line by interpolating the reference points suchthat the generated reference line is used as a reference line forcontrolling ADVs on the vehicular lane. In some embodiments, a referencepoints table and a road segments table representing the reference linesare downloaded in real-time to ADVs such that the ADVs can generatereference lines based on the ADVs' geographical location and drivingdirection. For example, in one embodiment, an ADV can generate areference line by requesting routing service for a path segment by apath segment identifier representing an upcoming road section aheadand/or based on the ADV's GPS location. Based on a path segmentidentifier, a routing service can return to the ADV reference pointstable containing reference points for all lanes of road segments ofinterest. ADV can look up reference points for a lane for a path segmentto generate a reference line for controlling the ADV on the vehicularlane.

FIG. 4A is a block diagram illustrating an example of a perceptionmodule 302 according to some embodiments. Referring to FIG. 4A,perception module 302 includes, but is not limited to, a sensorcomponent 411, a map component 412, and a history component 413. Thesemodules 411 through 413 may be implemented in software, hardware, or acombination thereof. The sensor component 411 may obtain sensor datafrom one or more sensors of an ADV. For example, the sensor component411 may periodically request or poll for sensor data from the one ormore sensors (e.g., may request sensor data from a sensor every fewmilliseconds, every second, or some other appropriate period of time).In another example, the sensor component 411 may listen or wait forsensor data to be received from the one or more sensors. For example,the sensor component 411 may be configured to constantly monitor buses,communication channels (wired or wireless), wires, lines, pins, traces,etc., so that the sensor component 411 is able to receive sensor data assoon as the sensor data is generated by the one or more sensors.

In one embodiment a sensor may be a camera (e.g., a digital camera, avideo camera, a video recorder, etc.) or some other device that iscapable of capturing or recording images. The sensor data generated bythe camera and received by the sensor component 411 may be referred toas video data. Examples of video data may include but are not limited todigital images (e.g., Joint Photographic Experts Group (JPEG) images),video frames, Motion Picture Experts Group (MPEG) data, or other datathat is appropriate for representing optical images captured by thecamera. In another embodiment, a sensor may be a radar unit (e.g., radarunit 214 illustrated in FIG. 2 ) or some other device that is capable ofdetermining the location, range, angle, and/or velocity of objectsaround the ADV using radio waves (e.g., radio-frequency waves orsignals). The sensor data generated by the radar unit may be referred toas radar data. Radar data may be data that may indicate the location,range, angle, and/or velocity of objects detected by the radar unit. Ina further embodiment, a sensor may be a LIDAR unit (e.g., LIDAR unit 215illustrated in FIG. 2 ) or some other device that is capable ofdetermining the location, range, angle, and/or velocity of objectsaround the ADV using light (e.g., laser light). The sensor datagenerated by the LIDAR unit may be data that may indicate the location,range, angle, and/or velocity of objects detected by the LIDAR unit. Inother embodiments, other types of sensors may generate other types ofsensor data which may be provided do the sensor component 111. Any typeof sensor that may be used to detect the location, range, angle, and/orvelocity of objects (e.g., pedestrians, vehicles, barricades, obstacles,barriers, lane lines, signs, traffic lights, etc.) in the environment orgeographical location/area may be used in the embodiments,implementations, and/or examples described here. In another embodiment,a sensor may be a GPS receiver or unit (e.g., GPS unit 212 illustratedin FIG. 2 ) or some other device capable to determining the location(e.g., physical or geographical location) of the ADV. The sensor datagenerated by the GPS receiver may be GPS data (which may be referred toas GPS coordinates).

In one embodiment, the sensor data may indicate information about theenvironment or geographical area/location in which the ADV is currentlylocated or travelling. For example, the sensor data may indicate thelocations and/or layouts of objects (e.g., pedestrians, vehicles,barricades, obstacles, barriers, lane lines, signs, traffic lights,etc.). In another example, the sensor data may indicate road conditionsfor the environment or geographical area (e.g., whether the road is adry road, wet road, smooth road, bumpy road, etc.). In a furtherexample, the sensor data may indicate weather conditions for theenvironment or geographical area (e.g., the temperature, whether thereis rain, wind, snow, hail, etc.).

In one embodiment, the history component 413 may obtain data indicativeof a path that was previously used by the ADV when the ADV previouslytravelled through the environment or geographical location/area. Forexample, the history component 413 may obtain GPS data (e.g., GPScoordinates, longitude/latitude coordinates, etc.) and may determine,based on the GPS data, whether the ADV has previously traveled throughthe environment or geographical location/area. If the history component413 determines that the ADV has previously travelled through theenvironment or geographical location/area, the history component 413 maydetermine a path that was previously used by the ADV to travel throughthe environment or geographical location/area. The history component 413may provide data indicative of the path that was previously used by theADV (e.g., GPS coordinates, etc.) to the decision module 304, planningmodule 305, and/or prediction module 303. The data (e.g., GPS data, GPScoordinates, longitude/latitude coordinates, etc.) indicative of a paththat was previously used by the ADV when the ADV previously travelledthrough the environment or geographical location/area may be referred toas path data.

In some embodiments, the map component 412 may generate map data, basedon the sensor data obtained/received by the sensor component 411. Themap data may indicate information about the environment or geographicallocation/area where the ADV may be travelling through or may be located.For example, the map data may indicate positions, locations,orientations, lengths, widths, distances, layouts, etc., of roads,lanes, signs (e.g., a stop sign, a yield sign, etc.), traffic lights,obstacles, buildings, sidewalks, pathways, walkways, barriers, etc. Inone embodiment, the map component 412 may generate the map data based onvideo data (e.g., sensor data) generated by or received from a camera(e.g., a sensor). For example, the map component 412 may analyze theimages or video (e.g., video data) captured by the camera to identifyroads, lanes, lane markers, sidewalks, lane dividers (e.g., a centerdivide), are located. In another example, the map component 412 mayanalyze radar data to identify objects, barriers, obstructions, roads,lanes, etc., in the environment or geographical area/location where theADV is located. In a further example, the map component 412 may analyzeLIDAR data to identify objects, barriers, obstructions, roads, lanes,etc., in the environment or geographical area/location where the ADV islocated.

In one embodiment, the map component 412 may generate map data based onthe path of other vehicles in the environment or geographicalarea/location. For example, the sensors of the ADV (e.g., cameras, radarunits, LIDAR units, etc.) may detect the path of another vehicle in theenvironment or geographical area/location. The map component 412 maydetermine the location, range, angle, and/or velocity of objects in theenvironment or geographical area/location based on the path of one ormore other vehicles. For example, the map component 412 may determinethe layout of roads, lands, etc., in the environment or geographicalarea/location where the ADV is located/travelling.

In one embodiment, the map component 412 may use various techniques,methods, algorithms, operations, etc., to generate the map data based onthe sensor data. For example, the map component 412 may use image orvideo processing/analysis techniques or algorithms to identify roads,lanes, buildings, etc., in the environment or geographical area/locationwhere the ADV is located, based on the video data. In another example,the map component 412 may use various object detection techniques oralgorithms to identify sidewalks, curbs, dividers (e.g., concrete landdividers), walls, etc., based on radar and/or LIDAR data. In a furtherexample, the map component 412 may use GPS data (e.g., GPS coordinates)to identify roads, lanes, highways, etc. The examples, implementations,and/or embodiments described may use various types of sensor data and/orvarious functions, techniques, methods, algorithms, operations, etc., togenerate the map data. For example, the map component 412 may usemachine learning, artificial intelligence, statistical models, neuralnetworks, clustering techniques, etc.

In one embodiment, the map component 412 may determine whether there isexisting map data for the environment and/or geographical area/locationwhere the ADV is currently travelling/moving through or is currentlylocated. For example, the map component 412 may determine that the ADVis current located in a particular environment or geographicalarea/location based on GPS data (e.g., based on GPS coordinates). Themap component 412 may determine that there is no preexisting map datafor the environment or geographical area/location. For example, the mapcomponent 412 may determine that map data for the environment orgeographical area/location was not previously downloaded or stored in apersistence storage device (e.g., persistent storage device 352illustrated in FIG. 3A). If there is no map data for the environment orgeographical area/location was not previously downloaded or stored in apersistence storage device, the map component 412 may generate the mapdata based on sensor data (e.g., video data, radar data, LIDAR data,etc.), as discussed above. The ADV may use the generated map data todetermine a path/route through the environment or geographicalarea/location based on the generated map data (e.g., the map datagenerated on the fly), as discussed in more detail below. If there ispreexisting map data for the environment or geographical area/location,the ADV may use the preexisting map data to determine a path/routethrough the environment or geographical area/location based on thepreexisting map data, as discussed in more detail below.

In some embodiments, the map component 412 may generate the map datawhen there is no preexisting map data for the environment orgeographical area/location as the ADV is travelling or moving throughthe environment or geographical area/location. This may be referred togenerating map data on the fly. For example, the map component 412 maygenerate map data for the environment or geographical area/location thatis in front of and/or around (e.g., surrounding) the ADV. For example,the map component 412 may generate map data for the area/location thatis fifty meters, one hundred meters, two hundred meters, or some otherappropriate distance, in front of the ADV. In another example, the mapcomponent 412 may generate map data for a circular area location aroundthe ADV (e.g., generate map data for a one hundred and fifty metercircle around the ADV with the ADV at the center of the circle).

In other embodiments, the size or distance of the area/location forwhich the map component 412 may generate map data on the fly may bebased on various factors such as the speed of the ADV, the distance/timeit may take the ADV to slow down or come to a complete stop, etc. Forexample, if the ADV is travelling at a higher speed/velocity, then thesize/distance of the area/location for which the map component 412 maygenerate map data, may be larger or longer. In another example, if theADV is travelling at a lower speed/velocity, then the size/distance ofthe area/location for which the map component 412 may generate map datamay be smaller or shorter.

Many of the hardware, software, firmware, algorithms, functions,methods, techniques, operations, etc., for determining a path for an ADVthrough an environment or geographical location/area use map data todetermine the path for the ADV. However, there may be situations,instances, or scenarios where the ADV does not have preexisting mapdata. For example, some ADVs may not have the storage space (e.g., harddrive space, disk space, memory space, etc.) to store preexisting mapdata or may not have any preexisting map data. In another example, theADV may travel to an environment or geographical area/location where theADV does not have preexisting map data (e.g., the ADV has map data forone state but does not have map data for another state). When map datais not available, the algorithms, functions, methods, techniques,operations, etc., for determining a path for the ADV may not be usablebecause they use the map data to determine the path. Generally, when anADV does not have map data for the environment or geographicalarea/location, the ADV may cease autonomous operation (e.g., the ADV maynot automatically drive through the environment or geographicalarea/location and may prompt a driver/user to take control of the ADV).Modifying the hardware, software, firmware, algorithms, functions,methods, techniques, operations, etc., to determining the path for theADV without using map data may be time consuming, error prone, and maynot be possible in some situations. Thus, it may be useful for an ADV tocontinue to operate automatically (e.g., automatically determine anddrive on a path through an environment or geographical area/location)even when map data (e.g., preexisting map data) is not available.

The embodiments, implementations, examples, etc., described herein allowthe ADV (e.g., the perception module 302, the map component 412, etc.)to generate map data based on sensor data generated by and/or obtainedfrom one or more sensors (e.g., a camera, a radar unit, a LIDAR unit,etc.) when there is no preexisting map data for an environment orgeographical area/location. This may allow the ADV to use existinghardware, software, firmware, algorithms, functions, methods,techniques, operations, etc., to determine a path for the ADV when thereis no preexisting map data. For example, instead of developingadditional hardware, software, firmware, algorithms, functions, methods,techniques, operations, etc., to determine a path for the ADV or ceasingautomatic operation of the ADV, the existing hardware, software,firmware, algorithms, functions, methods, techniques, operations, may beused because the map data is generated on the fly when no preexistingmap data exists.

FIG. 4B is a block diagram illustrating an example of a planning module305 according to some embodiments. Referring to FIG. 4B, planning module305 includes, but is not limited to, a segmenter 401, a quintic functiongenerator 402, a sample point generator 403, a path generator 404, and areference line generator 405. These modules 401-405 may be implementedin software, hardware, or a combination thereof. Reference linegenerator 405 is configured to generate a reference line for the ADV. Asdiscussed above, the reference line may be a guidance path, e.g., acenter line of a road, for the ADV, to generate stable trajectories. Thereference line generator 405 may generate the reference line based onmap and route information 311 (illustrated in FIGS. 3A and 3B). Asdiscussed above, the map and route information 311 may be preexistingmap data (e.g., map data that was previously downloaded or stored)and/or may be map data that is generated on the fly (e.g., map data foran area/location that is generated as the ADV travels through thearea/location). Segmenter 401 is configured to segment the referenceline into a number of reference line segments. The reference line may bedivided into reference line segments to generate discrete segments orportions of the reference line. For each of the reference line segments,polynomial function generator 402 may be configured to define andgenerate a polynomial function to represent or model the correspondingreference line segment. The sample point generator 403 may generatesample points based on the reference line. For example, the sample pointgenerator 403 may generate one or more sets of sample points (e.g.,groups of one or more sample points) that are may generally follow thereference line, as discussed in more detail below. Each set of samplepoints may include a first subset of sample points and a second subsetof sample points, as discussed in more detail below.

The polynomial function generator 402 may connect the multiple sets ofsample points to each other. For example, the polynomial functiongenerator 402 may generate one or more segments (e.g., connections)between each sample point in a set of sample points and each sample inthe next adjacent set of sample points, as discussed in more detailbelow. The polynomial function generator 402 may also generate,calculate, determine, etc., one or more polynomials that may be used torepresent the segments between the sample points. For example, thepolynomial function generator 402 may generate, determine, calculate,etc., a polynomial function for each segment between two sample points.The polynomial functions that represent the segments may also begenerated, determined, calculated based on various boundaries orconstraints. The boundaries or constraints may be preconfigured and/orstored as a part of constraints 313 illustrated in FIG. 3A. Thepolynomial functions used by the planning module 305 (e.g., used by thepolynomial function generator 402) may be preconfigured and/or stored asa part of functions polynomial functions 314 illustrated in FIG. 3A.

The path generator 404 may determine a path for the ADV based on thesegments between the sample points, as discussed in more detail below.For example, the path generator 404 may determine a cost for eachsegment. The cost may be based on various factors or parametersincluding, but not limited to, how far away the segment is from thereference line, how far away the sample points in the segment are fromthe reference line, the curvature change rate for a segment or forsample points in the segment, the curvature of a segment, obstacles(e.g., a vehicle, a pedestrian, an obstruction, etc.) that may belocated at a sample point, etc. The costs may also be referred to asweights. The path generator 404 may identify or select the segments thatform a path which has the lowest total cost (lowest total weight).

FIG. 5A is a diagram illustrating an example of an ADV 505 traveling(e.g., moving, driving, etc.) in an environment 500 (e.g., ageographical area/location) according to some embodiments. Theenvironment 500 includes a road 510 (e.g., a street, a lane, a highway,a freeway, an expressway, etc.) and a vehicle 515. The road 510 has twolanes, lane 511 and lane 512. The two lanes are divided by lane line 513and the boundaries of the road are defined by lane lines 516 and 517.The road 510, the lane lines 511, 512, 516, and 517, the location of theADV 505, the location of the vehicle 515, and/or other elementsillustrated in FIG. 5A may be represented using a Cartesian coordinatesystem as illustrated by the X-axis and Y-axis in FIG. 5A. For example,the location of the ADV 505 may be represented using an X-Y coordinate.

As discussed above, the ADV 505 may obtain sensor data from one or moresensors (e.g., a camera, a radar unit, a LIDAR unit, etc.). The sensordata may indicate the location, range, angle, and/or velocity of objectsin the environment 500 around the ADV 505. For example, the sensor datamay indicate the locations and/or layouts of the road 510 and the lanelines 511, 512, 516, and 517. The ADV 505 may also obtain dataindicative of a path 521 that was previously used by the ADV 505 whenthe ADV 505 previously travelled through the environment 500. The ADVmay also determine the path 531 used by the vehicle 515 based on thesensor data. The ADV 505 may generate map data, based on the sensordata, path 531 (used by the vehicle 515), and/or the path 521 (that waspreviously used by the ADV 505), as discussed above. The ADV 505 may usevarious techniques, methods, algorithms, operations, etc., to determinegenerate the map data based on the sensor data, as discussed above. Themap data may indicate information about the environment 500 (e.g., mayindicate positions, locations, orientations, lengths, widths, distances,etc., of roads, lanes, signs, traffic lights, obstacles, buildings,sidewalks, pathways, walkways, barriers, etc.).

The ADV 505 may determine whether there is preexisting map data for theenvironment 500. If there is preexisting map data for the environment500, the ADV may use the preexisting map data to determine a path forthe ADV 505 through the environment 500. If there is no preexisting mapdata for the environment 500, The ADV 505 may generate the map data(based on the sensor data) if there is preexisting map data for theenvironment 500. The ADV 505 may use the map data generated based on thesensor data (e.g., the map data generated on the fly) to determine apath for the ADV 505 through the environment 500. The ADV 505 maygenerate the map data as the ADV 505 is travelling or moving through theenvironment or geographical area/location. The ADV 505 may generate mapdata for the area/location in the environment 500 an appropriatedistance in front of and/or surrounding the ADV, based on variousfactors (e.g., the speed of the ADV 505), as discussed above.

FIG. 5B is a diagram illustrating an example of an ADV 505 traveling(e.g., moving, driving, etc.) in an environment 500 (e.g., ageographical area/location) according to some embodiments. Theenvironment 500 includes a road 510, lane lines 513, 516, and 517, and avehicle 515. As discussed above, reference line generator 405(illustrated in FIG. 4 ) may generate a reference line 530. Thereference line 530 may be a guidance path, e.g., a center line of theroad 530 for the ADV 505. Also as discussed above, segmenter 401(illustrated in FIG. 4 ) may segment (e.g., divide, split, etc.) thereference line 530 into reference line segments. The sample pointgenerator 503 may generate sample points 507 (illustrated by the blackdots in FIG. 5B), as discussed above. The sample points may be groupedinto groups or sets of sample points. As illustrated in FIG. 5B, thesample points 507 are grouped into three sets of sample points. The road510, the sample points, the reference line 530, and/or other elementsillustrated in FIG. 5 may be represented using a Cartesian coordinatesystem as illustrated by the X axis and Y-axis in FIG. 5B. For example,the location of the ADV 505 may be represented using an X-Y coordinate.In another example, a sample point may be represented using an X-Ycoordinate. In other embodiments, different numbers of reference linesegments, different numbers of sample points, different numbers of sets,different number of sample points in sets, different positions of samplepoints may be used.

In one embodiment, the reference line 530 may be represented using oneor more polynomial functions. For example, the polynomial functiongenerator 402 may generate a polynomial function that may representreference line segment 511 and a polynomial function that may representreference line segment. The polynomial function generator 402 maygenerate one polynomial function for each reference line segment. Foreach of the reference line segments, polynomial function generator 402may generate a polynomial function θ(s). In one embodiment, eachpolynomial function represents a direction of a starting reference pointof the corresponding reference line segment. A derivative (e.g., thefirst order derivative) of the polynomial function represents acurvature of the starting reference point of the reference line segment,K=dθ/ds. A second order derivative of the polynomial function representsa curvature change or curvature change rate, dK/ds.

For the purpose of illustration, following terms are defined:

-   -   θ₀: starting direction    -   {dot over (θ)}₀: starting curvature, κ, direction derivative        w.r.t. curve length, i.e.,

$\frac{d\;\theta}{ds}$

-   -   {umlaut over (θ)}₀: starting curvature derivative, i.e.,

$\frac{d\;\kappa}{ds}$

-   -   θ₁: ending direction    -   {dot over (θ)}₁: ending direction    -   {umlaut over (θ)}₁: ending curvature derivative    -   Δs: the curve length between the two ends

Each piecewise spiral path is decided by seven parameters: startingdirection (θ0), starting curvature (dθ0), starting curvature derivative(d2θ0), ending direction (θ1), ending curvature (dθ1), ending curvaturederivative (d2θ1) and the curve length between the starting and endingpoints (Δs). In one embodiment, the polynomial function may be a quinticpolynomial function. A quintic polynomial function may be defined byequation (1) (e.g., a formula, a function, etc.) as follows:θ_(i)(s)=a*s ⁵ +b*s ⁴ +c*s ³ +d*s ² +e*s+f  (1)and it satisfiesθ_(i)(0)=θ_(i)  (2){dot over (θ)}_(i)(0)={dot over (θ)}_(i)  (3){umlaut over (θ)}_(i)(0)={umlaut over (θ)}_(i)  (4)θ_(i)(Δs)=θ_(i+1)  (5){dot over (θ)}_(i)(Δs)={dot over (θ)}_(i+1)  (6){umlaut over (θ)}_(i)(Δs)={umlaut over (θ)}_(i+1)  (7)

In another embodiment, the polynomial function may be a cubicpolynomial. A cubic polynomial may be defined by equation (8) asfollows:θ_(i)(s)=a*s ³ +b*s ² +c*s+f  (8)and the cubic polynomial may satisfy the same conditions (indicatedabove with respect to the quintic polynomial function) illustrated byequations (2) through (7).

Based on the above constraints, the optimization is performed on allpolynomial functions of all reference line segments, such that theoutput of a polynomial function representing reference line segment (i)at zero segment length should be the same as or similar to a directionat the starting reference point of the corresponding reference linesegment (i). A first order derivative of the polynomial function shouldbe the same as or similar to a curvature at the starting reference pointof the reference line segment (i). A second order derivative of thepolynomial function should be the same as or similar to a curvaturechange rate at the starting reference point of the reference linesegment (i). Similarly, the output of a polynomial function representingreference line segment (i) at the full segment length (s) should be thesame as or similar to a direction at the starting reference point of thenext reference line segment (i+1), which is the ending reference pointof the current reference line segment (i). A first order derivative ofthe polynomial function should be the same as or similar to a curvatureat the starting reference point of the next reference line segment(i+1). A second order derivative of the polynomial function should bethe same as or similar to a curvature change rate at the startingreference point of the next reference line segment (i+1).

For example, for a reference line segment, an output of thecorresponding polynomial function θ(0) represents a direction or angleof a starting point of reference line segment. θ(Δs0) represents adirection of ending point of reference line segments, where the endingpoint of reference line segments is also the starting point of the nextreference line segment. A first order derivative of θ(0) represents acurvature at the starting point of reference line segment and a secondorder derivative of θ(0) represents a curvature change rate at theending point of reference line segment. A first order derivative ofθ(s0) represents a curvature of the ending point of reference linesegment and a second order derivative of θ(s0) represents a curvaturechange rate of the ending point of reference line segment.

By substituting the above variables θ_(i), {dot over (θ)}_(i), {umlautover (θ)}_(i), θ_(i+1), {dot over (θ)}_(i+1), {umlaut over (θ)}_(i+1),Δs in, there will be six equations that may be utilized to solve thecoefficients of the polynomial function a, b, c, d, e, and f. Forexample, as stated above, the direction at a given point may be definedusing the above quintic polynomial function:θ(s)=as ⁵ +bs ⁴ +cs ³ +ds ² +es+f  (9)

The first order derivative of the quintic polynomial function representsa curvature at the point of the path:dθ=5as ⁴+4bs ³+3cs ²+2ds+e  (10)

The second order derivative of the quintic polynomial functionrepresents a curvature change rate at the point of the path:d ²θ=20as ³+12bs ²+6cs+2d  (11)

For a given spiral path or reference line segment, there are two pointsinvolved: a starting point and an ending point, where the direction,curvature, and curvature change rate of each point may be represented bythe above three equations respectively. Thus, there are a total of sixequations for each spiral path or reference line segment. These sixequations may be utilized to determine the coefficients a, b, c, d, e,and f of the corresponding quintic polynomial function.

When a spiral path is utilized to represent a curve between consecutivereference points in the Cartesian space, there is a need to build aconnection or bridge between the spiral path curve length and a positionin the Cartesian space. Given a spiral path θ_(i)(s) defined by {θ_(i),dθ_(i), d²θ_(i), θ_(i+1), dθ_(i+1), d²θ_(i+1), Δs}, and path startingpoint p_(i)=(x_(i), y_(i)), we need to determine the coordinate of pointp=(x, y) given any s=[0, Δs]. In one embodiment, the coordinates of agiven point may be obtained based on the following equations (e.g.,formulas, functions, etc.):x=x _(i)+∫₀ ^(s) cos(θi(s))ds  (12)y=y _(i)+∫₀ ^(s) cos(θi(s))ds  (13)

When s=Δs, the ending coordinates pi+1 are obtained given curve θi andstarting coordinates pi=(xi, yi). The optimization of the functions areperformed such that the overall output of the functions of the spiralpaths reach minimum, while the above set of constraints are satisfied.In addition, the coordinates of the terminal point derived from theoptimization is required to be within a predetermined range (e.g.,tolerance, error margins) with respect to the corresponding coordinatesof the initial reference line. That is, the difference between eachoptimized point and the corresponding point of the initial referenceline should be within a predetermined threshold.

In some embodiments, the path generator 404 may use dynamic programmingalgorithm, functions, operations, etc., to determine the path for theADV 505. For example, the path generator 404 may use Dijkstra'salgorithm to determine the path with the lowest cost for the ADV 505based on the costs (e.g., the weights) of the segments. The path for theADV may include one segment of the nine segments between the leftmostset of sample points and the middle set of sample points, and onesegment of the nine segments between the middle set of sample points andthe rightmost set of sample points. If multiple paths have the samelowest cost, the path generator 404 may select one of the multiple pathsbased on various factors. For example, the path generator 404 may selectthe path that most closely follows the reference line 530 (e.g., thepath that deviates the least from the reference line 530).

FIG. 6 is a flow diagram illustrating an example of process 600 fordetermining a path for an autonomous vehicle according to someembodiments. Process 600 may be performed by processing logic which mayinclude software, hardware, or a combination thereof. Process 600 may beperformed by processing logic that may comprise hardware (e.g.,circuitry, dedicated logic, programmable logic, a processor, aprocessing device, a central processing unit (CPU), a system-on-chip(SoC), etc.), software (e.g., instructions running/executing on aprocessing device), firmware (e.g., microcode), or a combinationthereof. In some embodiments, process 600 may be performed by one ormore of perception module 302 and planning module 305, illustrated inFIG. 3B, FIG. 4A, and FIG. 4B. Referring to FIG. 6 , at block 605,processing logic obtains sensor data (e.g., video data, image data,radar data, LIDAR data, GPS data, etc.) from one or more sensors (e.g.,a camera, a radar unit, a LIDAR unit, a GPS unit, etc.). For example,the processing logic may request the sensor data from the one or moresensors or the processing logic may wait (e.g., listen) for sensor datafrom the one or more sensors. At block 610, the processing logic maydetermine whether there is preexisting map data for the environment orgeographical area/location where an ADV is located/travelling. Forexample, the processing logic may determine the GPS coordinates of theADV and may determine whether there is preexisting map data based on theGPS coordinates.

If there is preexisting map data for the environment or geographicalarea/location where the ADV is located/travelling, the processing logicmay determine a path for the ADV based on the preexisting map data atblock 625. For example, the processing logic may determine a referenceline, sample points, and segments for a path, based on the preexistingmap data. If there is no preexisting map data the processing logic maygenerate the map data based on the sensor data at block 615. Forexample, the processing logic may identify the locations or roads,lines, sidewalks, buildings, obstacles, etc., based on video data, radardata, LIDAR data, etc., as discussed above. The map data may indicatethe location, range, angle, and/or velocity of objects in theenvironment or geographical area/location around the ADV. The size ordistance of the area/location for which the processing logic maygenerate map data on the fly may be based on various factors such as thespeed of the ADV. At block 620, the processing logic may determine apath for the ADV based on the generated map data (e.g., the map datagenerated based on the sensor data in response to determining that thereis no preexisting map data for the environment or geographicalarea/location). For example, the processing logic may determine areference line, sample points, and segments for a path, based on thegenerated map data. At block 630, the processing logic may control theADV based on the path. For example, the processing logic may cause theADV to move along the path (e.g., may steer the ADV to drive along thepath).

Note that some or all of the components as shown and described above maybe implemented in software, hardware, or a combination thereof. Forexample, such components may be implemented as software installed andstored in a persistent storage device, which may be loaded and executedin a memory by a processor (not shown) to carry out the processes oroperations described throughout this application. Alternatively, suchcomponents may be implemented as executable code programmed or embeddedinto dedicated hardware such as an integrated circuit (e.g., anapplication specific IC or ASIC), a digital signal processor (DSP), or afield programmable gate array (FPGA), which may be accessed via acorresponding driver and/or operating system from an application.Furthermore, such components may be implemented as specific hardwarelogic in a processor or processor core as part of an instruction setaccessible by a software component via one or more specificinstructions.

FIG. 7 is a block diagram illustrating an example of a data processingsystem which may be used with one embodiment of the disclosure. Forexample, system 1500 may represent any of data processing systemsdescribed above performing any of the processes or methods describedabove, such as, for example, perception and planning system 110 or anyof servers 103-104 of FIG. 1 . System 1500 can include many differentcomponents. These components may be implemented as integrated circuits(ICs), portions thereof, discrete electronic devices, or other modulesadapted to a circuit board such as a motherboard or add-in card of thecomputer system, or as components otherwise incorporated within achassis of the computer system.

Note also that system 1500 is intended to show a high level view of manycomponents of the computer system. However, it is to be understood thatadditional components may be present in certain implementations andfurthermore, different arrangement of the components shown may occur inother implementations. System 1500 may represent a desktop, a laptop, atablet, a server, a mobile phone, a media player, a personal digitalassistant (PDA), a Smartwatch, a personal communicator, a gaming device,a network router or hub, a wireless access point (AP) or repeater, aset-top box, or a combination thereof. Further, while only a singlemachine or system is illustrated, the term “machine” or “system” shallalso be taken to include any collection of machines or systems thatindividually or jointly execute a set (or multiple sets) of instructionsto perform any one or more of the methodologies discussed herein.

In one embodiment, system 1500 includes processor 1501, memory 1503, anddevices 1505-1508 connected via a bus or an interconnect 1510. Processor1501 may represent a single processor or multiple processors with asingle processor core or multiple processor cores included therein.Processor 1501 may represent one or more general-purpose processors suchas a microprocessor, a central processing unit (CPU), or the like. Moreparticularly, processor 1501 may be a complex instruction set computing(CISC) microprocessor, reduced instruction set computing (RISC)microprocessor, very long instruction word (VLIW) microprocessor, orprocessor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processor 1501 may alsobe one or more special-purpose processors such as an applicationspecific integrated circuit (ASIC), a cellular or baseband processor, afield programmable gate array (FPGA), a digital signal processor (DSP),a network processor, a graphics processor, a communications processor, acryptographic processor, a co-processor, an embedded processor, or anyother type of logic capable of processing instructions.

Processor 1501, which may be a low power multi-core processor socketsuch as an ultra-low voltage processor, may act as a main processingunit and central hub for communication with the various components ofthe system. Such processor may be implemented as a system on chip (SoC).Processor 1501 is configured to execute instructions for performing theoperations and steps discussed herein. System 1500 may further include agraphics interface that communicates with optional graphics subsystem1504, which may include a display controller, a graphics processor,and/or a display device.

Processor 1501 may communicate with memory 1503, which in one embodimentmay be implemented via multiple memory devices to provide for a givenamount of system memory. Memory 1503 may include one or more volatilestorage (or memory) devices such as random access memory (RAM), dynamicRAM (DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), or other typesof storage devices. Memory 1503 may store information includingsequences of instructions that are executed by processor 1501, or anyother device. For example, executable code and/or data of a variety ofoperating systems, device drivers, firmware (e.g., input output basicsystem or BIOS), and/or applications may be loaded in memory 1503 andexecuted by processor 1501. An operating system may be any kind ofoperating systems, such as, for example, Robot Operating System (ROS),Windows® operating system from Microsoft®, Mac OS®/iOS® from Apple,Android® from Google®, LINUX, UNIX, or other real-time or embeddedoperating systems.

System 1500 may further include 10 devices such as devices 1505-1508,including network interface device(s) 1505, optional input device(s)1506, and other optional IO device(s) 1507. Network interface device1505 may include a wireless transceiver and/or a network interface card(NIC). The wireless transceiver may be a WiFi transceiver, an infraredtransceiver, a Bluetooth transceiver, a WiMax transceiver, a wirelesscellular telephony transceiver, a satellite transceiver (e.g., a globalpositioning system (GPS) transceiver), or other radio frequency (RF)transceivers, or a combination thereof. The NIC may be an Ethernet card.

Input device(s) 1506 may include a mouse, a touch pad, a touch sensitivescreen (which may be integrated with display device 1504), a pointerdevice such as a stylus, and/or a keyboard (e.g., physical keyboard or avirtual keyboard displayed as part of a touch sensitive screen). Forexample, input device 1506 may include a touch screen controller coupledto a touch screen. The touch screen and touch screen controller can, forexample, detect contact and movement or break thereof using any of aplurality of touch sensitivity technologies, including but not limitedto capacitive, resistive, infrared, and surface acoustic wavetechnologies, as well as other proximity sensor arrays or other elementsfor determining one or more points of contact with the touch screen.

IO devices 1507 may include an audio device. An audio device may includea speaker and/or a microphone to facilitate voice-enabled functions,such as voice recognition, voice replication, digital recording, and/ortelephony functions. Other IO devices 1507 may further include universalserial bus (USB) port(s), parallel port(s), serial port(s), a printer, anetwork interface, a bus bridge (e.g., a PCI-PCI bridge), sensor(s)(e.g., a motion sensor such as an accelerometer, gyroscope, amagnetometer, a light sensor, compass, a proximity sensor, etc.), or acombination thereof. Devices 1507 may further include an imagingprocessing subsystem (e.g., a camera), which may include an opticalsensor, such as a charged coupled device (CCD) or a complementarymetal-oxide semiconductor (CMOS) optical sensor, utilized to facilitatecamera functions, such as recording photographs and video clips. Certainsensors may be coupled to interconnect 1510 via a sensor hub (notshown), while other devices such as a keyboard or thermal sensor may becontrolled by an embedded controller (not shown), dependent upon thespecific configuration or design of system 1500.

To provide for persistent storage of information such as data,applications, one or more operating systems and so forth, a mass storage(not shown) may also couple to processor 1501. In various embodiments,to enable a thinner and lighter system design as well as to improvesystem responsiveness, this mass storage may be implemented via a solidstate device (SSD). However in other embodiments, the mass storage mayprimarily be implemented using a hard disk drive (HDD) with a smalleramount of SSD storage to act as a SSD cache to enable non-volatilestorage of context state and other such information during power downevents so that a fast power up can occur on re-initiation of systemactivities. Also a flash device may be coupled to processor 1501, e.g.,via a serial peripheral interface (SPI). This flash device may providefor non-volatile storage of system software, including BIOS as well asother firmware of the system.

Storage device 1508 may include computer-accessible storage medium 1509(also known as a machine-readable storage medium or a computer-readablemedium) on which is stored one or more sets of instructions or software(e.g., module, unit, and/or logic 1528) embodying any one or more of themethodologies or functions described herein. Processingmodule/unit/logic 1528 may represent any of the components describedabove, such as, for example, planning module 305, control module 306.Processing module/unit/logic 1528 may also reside, completely or atleast partially, within memory 1503 and/or within processor 1501 duringexecution thereof by data processing system 1500, memory 1503 andprocessor 1501 also constituting machine-accessible storage media.Processing module/unit/logic 1528 may further be transmitted or receivedover a network via network interface device 1505.

Computer-readable storage medium 1509 may also be used to store the somesoftware functionalities described above persistently. Whilecomputer-readable storage medium 1509 is shown in an exemplaryembodiment to be a single medium, the term “computer-readable storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more sets of instructions. The terms“computer-readable storage medium” shall also be taken to include anymedium that is capable of storing or encoding a set of instructions forexecution by the machine and that cause the machine to perform any oneor more of the methodologies of the present disclosure. The term“computer-readable storage medium” shall accordingly be taken toinclude, but not be limited to, solid-state memories, and optical andmagnetic media, or any other non-transitory machine-readable medium.

Processing module/unit/logic 1528, components and other featuresdescribed herein may be implemented as discrete hardware components orintegrated in the functionality of hardware components such as ASICS,FPGAs, DSPs or similar devices. In addition, processingmodule/unit/logic 1528 may be implemented as firmware or functionalcircuitry within hardware devices. Further, processing module/unit/logic1528 may be implemented in any combination hardware devices and softwarecomponents.

Note that while system 1500 is illustrated with various components of adata processing system, it is not intended to represent any particulararchitecture or manner of interconnecting the components; as suchdetails are not germane to embodiments of the present disclosure. Itwill also be appreciated that network computers, handheld computers,mobile phones, servers, and/or other data processing systems which havefewer components or perhaps more components may also be used withembodiments of the disclosure.

Some portions of the preceding detailed descriptions have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as those set forth in the claims below, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the disclosure also relate to an apparatus for performingthe operations herein. Such a computer program is stored in anon-transitory computer readable medium. A machine-readable mediumincludes any mechanism for storing information in a form readable by amachine (e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices).

The processes or methods depicted in the preceding figures may beperformed by processing logic that comprises hardware (e.g. circuitry,dedicated logic, etc.), software (e.g., embodied on a non-transitorycomputer readable medium), or a combination of both. Although theprocesses or methods are described above in terms of some sequentialoperations, it should be appreciated that some of the operationsdescribed may be performed in a different order. Moreover, someoperations may be performed in parallel rather than sequentially.

Embodiments of the present disclosure are not described with referenceto any particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof embodiments of the disclosure as described herein.

In the foregoing specification, embodiments of the disclosure have beendescribed with reference to specific exemplary embodiments thereof. Itwill be evident that various modifications may be made thereto withoutdeparting from the broader spirit and scope of the disclosure as setforth in the following claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

What is claimed is:
 1. A computer-implemented method for operating anautonomous driving vehicle, the method comprising: obtaining sensordata, wherein the sensor data indicates information about an environmentthrough which the autonomous driving vehicle is currently travelingbased on a selected route for the autonomous driving vehicle;determining that preexisting map data of the environment is notavailable for determining a path of the autonomous driving vehiclethrough the environment; in response to determining that preexisting mapdata of the environment is not available, adjusting operation of theautonomous driving vehicle from using preexisting map data to determinethe path of the autonomous driving vehicle to generating and usinglocally generated map data to determine the path of the autonomousdriving vehicle, wherein generating the locally generated map datacomprises: determining an area of the environment around the autonomousdriving vehicle to map including determining a size of the area based onone or more factors, the one or more factors comprising at least a speedof the autonomous driving vehicle; identifying a location, range, angle,and velocity of one or more objects within the area based on the sensordata, wherein the one or more objects comprise roads, road lines,sidewalks, buildings, and obstacles; and generating, by a processingdevice of the autonomous driving vehicle, current map data of the areaof the environment around the autonomous driving vehicle based on thesensor data and the one or more objects as the autonomous drivingvehicle travels through the environment, the current map data to be usedin place of the preexisting map data for determining a path of theautonomous driving vehicle through the environment; determining areference line, sample points, and segments for a path of the autonomousdriving vehicle based on the current map data generated locally by theprocessing device of the autonomous driving vehicle and the selectedroute for the autonomous driving vehicle; and controlling the autonomousdriving vehicle based on the reference line, sample points, and segmentsfor the path of the autonomous driving vehicle.
 2. The method of claim1, further comprising: determining whether there is existing map datafor the environment.
 3. The method of claim 2, wherein the current mapdata is generated in response to determining that there is no existingmap data.
 4. The method of claim 2, further comprising: determining apath for the autonomous driving vehicle based on the existing map datain response to determining that there is existing map data.
 5. Themethod of claim 1, further comprising: obtaining path data indicative ofa previous path used by the autonomous driving vehicle when theautonomous driving vehicle previously traveled through the environment.6. The method of claim 5, wherein the path for the autonomous drivingvehicle is determined further based on the previous path.
 7. The methodof claim 1, wherein the sensor data is generated by one or more sensorsof the autonomous driving vehicle.
 8. The method of claim 7, wherein thesensor data comprises video data and wherein the one or more sensorscomprise a camera.
 9. The method of claim 7, wherein the sensor datacomprises radar data and wherein the one or more sensors comprise aradar unit.
 10. The method of claim 7, wherein the sensor data compriseslight detection and range (LIDAR) data and wherein the one or moresensors comprises a light detection and range unit.
 11. A non-transitorymachine-readable medium having instructions stored therein, which whenexecuted by a processor, cause the processor to perform operations, theoperations comprising: obtaining sensor data, wherein the sensor dataindicates information about an environment through which an autonomousdriving vehicle is currently traveling based on a selected route for theautonomous driving vehicle; determining that preexisting map data of theenvironment is not available for determining a path of the autonomousdriving vehicle through the environment; in response to determining thatpreexisting map data of the environment is not available, adjustingoperation of the autonomous driving vehicle from using preexisting mapdata to determine the path of the autonomous driving vehicle togenerating and using locally generated map data to determine the path ofthe autonomous driving vehicle, wherein generating the locally generatedmap data comprises: determining an area of the environment around theautonomous driving vehicle to map including determining a size of thearea based on one or more factors, the one or more factors comprising atleast a speed of the autonomous driving vehicle; identifying a location,range, angle, and velocity of one or more objects within the area basedon the sensor data, wherein the one or more objects comprise roads, roadlines, sidewalks, buildings, and obstacles; and generating current mapdata of the area of the environment around the autonomous drivingvehicle based on the sensor data and the one or more objects as theautonomous driving vehicle travels through the environment, the currentmap data to be used in place of the preexisting map data for determininga path of the autonomous driving vehicle; determining a reference line,sample points, and segments for a path of the autonomous driving vehiclebased on the current map data generated locally by a processing deviceof the autonomous driving vehicle and the selected route for theautonomous driving vehicle; and controlling the autonomous drivingvehicle based on the reference line, sample points, and segments for thepath of the autonomous vehicle.
 12. The non-transitory machine-readablemedium of claim 11, wherein the operations further comprise: determiningwhether there is existing map data for the environment.
 13. Thenon-transitory machine-readable medium of claim 12, wherein the currentmap data is generated in response to determining that there is noexisting map data.
 14. The non-transitory machine-readable medium ofclaim 12, wherein the operations further comprise: determining a pathfor the autonomous driving vehicle based on the existing map data inresponse to determining that there is existing map data.
 15. Thenon-transitory machine-readable medium of claim 11, wherein theoperations further comprise: obtaining a previous path used by theautonomous driving vehicle when the autonomous driving vehiclepreviously traveled through the environment.
 16. The non-transitorymachine-readable medium of claim 15, wherein the path for the autonomousdriving vehicle is determined further based on the previous path. 17.The non-transitory machine-readable medium of claim 11, wherein thesensor data is generated by one or more sensors of the autonomousdriving vehicle.
 18. A data processing system, comprising: a processor;and a memory coupled to the processor to store instructions, which whenexecuted by the processor, cause the processor to perform operations,the operations comprising: obtaining sensor data, wherein the sensordata indicates information about an environment through which anautonomous driving vehicle is currently traveling based on a selectedroute for the autonomous driving vehicle; determining that preexistingmap data of the environment is not available for determining a path ofthe autonomous driving vehicle through the environment; in response todetermining that preexisting map data of the environment is notavailable, adjusting operation of the autonomous driving vehicle fromusing preexisting map data to determine the path of the autonomousdriving vehicle to generating and using locally generated map data todetermine the path of the autonomous driving vehicle, wherein generatingthe locally generated map data comprises: determining an area of theenvironment around the autonomous driving vehicle to map includingdetermining a size of the area based on one or more factors, the one ormore factors comprising at least a speed of the autonomous drivingvehicle; identifying a location, range, angle, and velocity of one ormore objects within the area based on the sensor data, wherein the oneor more objects comprise roads, road lines, sidewalks, buildings, andobstacles; and generating current map data of the area of theenvironment around the autonomous driving vehicle based on the sensordata and the one or more objects as the autonomous driving vehicletravels through the environment, the current map data to be used inplace of the preexisting map data for determining a path of theautonomous driving vehicle through the environment; determining areference line, sample points, and segments for a path of the autonomousdriving vehicle based on the current map data generated locally by aprocessing device of the autonomous driving vehicle and the selectedroute for the autonomous driving vehicle; and controlling the autonomousdriving vehicle based on the reference line, sample points, and segmentsfor the path of the autonomous driving vehicle.